Skip to content

fix(guard): close the two live holes in obligations (c) and (f) - #408

Merged
avrabe merged 4 commits into
mainfrom
fix/gate-evidence-batch2
Aug 7, 2026
Merged

fix(guard): close the two live holes in obligations (c) and (f)#408
avrabe merged 4 commits into
mainfrom
fix/gate-evidence-batch2

Conversation

@avrabe

@avrabe avrabe commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Batch 2 of the multi-persona review of REQ-GUARD-GATE-EVIDENCE-002. Closes #404,
refs #385 (defect b). Follows #407.

Both defects were live on main, and in both the earlier fix had closed the
adjacent question rather than the reported one.

(f) #385 defect (b): the pattern was as narrow as the exemption it replaced

The first revision fixed the self-service -- TODO exemption — defect (a) — and
left _SORRY as the old gate's regex verbatim. So the four forms #385 named
explicitly all reported zero:

form before after
sorry alone on its line 1 1
theorem cheat : 1 = 2 := by sorry 0 1
theorem cheat : 1 = 2 := sorry 0 1
admit 0 1
axiom cheat : 7 = 8 0 1

An author could assert a falsehood as an axiom and the declared floor never
moved. Worse, the self-test contained a case asserting that a non-bare
occurrence does not count — pinning the hole shut rather than finding it, which
makes a gap look deliberate. That case is gone; each form now has one requiring
it to count.

Detection runs over comment-stripped text (nested /- -/ plus --), which is
what allows word-bounded patterns instead of line anchors: prose mentioning
sorry still does not count, and cases pin that too.

MEASURED before widening — the count is 12 either way. The tree contains no
inline sorry, no admit, no axiom and no sorryAx today, so this is a pure
tightening: no floor change, no latent red, only the hatches closed. Output is
now broken out by kind, so a shift from sorry to axiom at a constant total
becomes visible.

(c) #404: the gate parsed -p and then ignored it

extract_filter extracted the package at line 165; the vacuity test at line 212
matched against a flat workspace-wide inventory. The gate asked "does this
substring occur anywhere?"
while run_verification.py runs "does it occur in
package P?"
— strictly weaker, and #388's own executed proof case survived
it
: cargo test -p spar-wasm -- topology matches three tests, all of them in
spar-solver and spar-network.

The inventory is now collected per package (10 listings sharing one build) and
each filter judged against its own. A package that cannot be inventoried is
CANNOT-JUDGE (exit 2), not a pass. --inventory-json now requires an object —
a flat array is rejected, because a flat list cannot express which package owns
a test, which is the defect itself.

TEST-STPA-SVG-TOPOLOGY is demoted to proposed, not repointed. It
describes eight tests covering labels and connection endpoints; spar-wasm's
nearest candidates are the four in graph::tests. The described evidence cannot
be located, and choosing a filter that merely selects something would assert a
relevance nobody has checked. A human should decide whether the eight tests get
written or the claim gets withdrawn.

Evidence

  • 16 cases (sorries), 18 cases (filters), all passing; live sorry gate reports
    obligations found: 12 (declared floor: 12).
  • Mutation-tested, all mutants caught, including the exact one the
    adversarial reviewer used to falsify the old blanket claim:
    filtfilt.split('::')[-1], which every previous case was blind to
    because none carried a :: while 16 of the 60 real filters do.
  • Three tools have now needed the same repair — pin the diagnosis, not just
    the exit code.
    Distinct broken inputs reach the same exit code by different
    routes (a missing file raises; a missing key reads as None; an
    un-inventoried package and an empty inventory both exit 2), so removing a
    guard left the suites green. Defence-in-depth and testability pull against
    each other, and the resolution is to assert which branch spoke.

That recurrence is the argument for #405's potency harness: mutations run by
hand found survivors in three of five tools this session, and running them by
hand is precisely the unrepeatable practice #407 withdrew a claim about.

Not in this PR

REQ-GUARD-GATE-EVIDENCE-002 stays proposed with its human-scoped tag.

🤖 Generated with Claude Code

avrabe and others added 2 commits August 7, 2026 12:20
Batch 2 of the multi-persona review of REQ-GUARD-GATE-EVIDENCE-002. Both
fixes address a defect that was LIVE on main, and in both cases the previous
fix had closed the adjacent question.

(f) #385 defect (b): the sorry pattern was as narrow as the exemption

The first revision fixed the self-service `-- TODO` exemption (defect a) and
left the pattern untouched, so `:= by sorry`, `:= sorry`, `admit` and
`axiom cheat : 7 = 8` all reported zero — the four forms the issue named
explicitly. An author could assert a falsehood and the floor never moved.
Worse, the self-test contained a case ASSERTING that a non-bare occurrence
does not count, which made the hole look deliberate.

Detection now runs over comment-stripped text (nested `/- -/` and `--`),
which is what allows word-bounded patterns instead of line anchors: prose
mentioning sorry still does not count, and every real form does. `sorryAx`
is counted separately since `\bsorry\b` cannot see inside it.

MEASURED before widening: the count is 12 either way. The tree contains no
inline sorry, no admit, no axiom, no sorryAx today, so this is a pure
tightening with no floor change and no latent red. Output is now broken out
by kind, so a shift from `sorry` to `axiom` at a constant total is visible.

(c) #404: the filter gate parsed `-p` and then ignored it

`extract_filter` extracted the package; the vacuity test matched against a
flat workspace-wide inventory. So the gate asked "does this substring occur
anywhere?" while run_verification.py runs "does it occur in package P?" —
strictly weaker, and #388's own executed proof case survived it.
`cargo test -p spar-wasm -- topology` matches three tests, all in
spar-solver and spar-network.

The inventory is now collected per package and each filter judged against
its own. A package that cannot be inventoried is CANNOT-JUDGE (exit 2), not
a pass. `--inventory-json` now requires an object; a flat array is rejected
because a flat list cannot express which package owns a test, which is the
defect itself.

TEST-STPA-SVG-TOPOLOGY is demoted to `proposed` rather than repointed. Its
description claims eight tests covering labels and connection endpoints;
spar-wasm's nearest candidates are the four in `graph::tests`. The described
evidence cannot be located, and picking a filter that merely selects
something would assert a relevance nobody has checked.

Both suites mutation-tested, all mutants caught. Two needed the same fix as
check_mutants_report.py — pinning the DIAGNOSIS, not just the exit code —
because distinct broken inputs reach the same exit code by different routes,
so removing a guard left the suite green. That includes the exact mutant the
adversarial reviewer used to falsify the old mutation-testing claim:
`filt` -> `filt.split('::')[-1]`, which no case could see because none had a
`::` filter while 16 of 60 real ones do.

Closes #404. Refs #385 (defect b). REQ-GUARD-GATE-EVIDENCE-002 (c), (f).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

Rivet verification gate

20/20 passed

count
Passed 20
Failed 0
Skipped (no steps) 0

Filter: (and (= type "feature") (or (has-tag "v093") (has-tag "v0100")))

Failed artifacts

(none)

Updated automatically by tools/post_verification_comment.py. Source of truth: artifacts/verification.yaml.

@codecov

codecov Bot commented Aug 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

avrabe and others added 2 commits August 7, 2026 17:04
Found by the per-package inventory the previous commit introduced — its
first contact with a real cargo workspace, and it failed closed as designed.

`crates/spar-cli/Cargo.toml` declares `name = "spar"`. There is no package
called `spar-cli` anywhere in the workspace, and four verification steps
said `cargo test -p spar-cli`:

    TEST-CLI          cargo test -p spar-cli
    VAL-VERIFY-001    cargo test -p spar-cli -- verify
    VAL-DIFF-001      cargo test -p spar-cli -- diff
    VAL-MCP-001       cargo test -p spar-cli -- mcp

cargo answers `error: package ID specification 'spar-cli' did not match any
packages`, so every one of those steps has always failed when actually run.
The old flat inventory never noticed because it never passed `-p` to cargo
at all — the same discarded-package defect as #404, showing up as a broken
reference rather than a mis-scoped one.

Rewritten to `-p spar`. The filters have candidates under crates/spar-cli/
(verify 24, diff 2, mcp 2), and three of the four are `proposed` in any case.

Also: a package cargo cannot resolve is now a finding ABOUT those artifacts
rather than a traceback. `collect_inventory` distinguishes "did not match any
packages" (omit it, let check() name each affected artifact and exit 2) from
any other cargo failure (still fatal — a broken build invalidates every
verdict, not one artifact's). A traceback names only the first offender;
this names all of them.

19 self-test cases, including one asserting the artifact and package are
named rather than merely exited on.

Refs #404. REQ-GUARD-GATE-EVIDENCE-002 (c).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… tests be an answer

Two faults in the per-package collection, both mine, both found by CI on
real data.

1. It inventoried packages no filter asks about

`main` gathered every package named by any step, including whole-package
steps (`cargo test -p X` with no `--`). Those select everything and can
never be vacuous, so the inventory is work nothing consumes — and it made
the gate FATAL on `spar-verify-macros`, a proc-macro crate with no tests
that no filter mentions. Now only packages of steps that actually carry a
filter are listed: 9 instead of 10, with eleven whole-package-only crates
(spar-annex, spar-dbc, spar-hir, spar-insight, spar-mcp, spar-mermaid,
spar-parser, spar-solver, spar-transform, spar-variants,
spar-verify-macros) no longer built for nothing.

2. Zero tests was treated as a broken scan

An empty list from a SUCCESSFUL cargo run is a truthful answer, not a
failure — some packages genuinely have no tests. Keeping the empty set is
also the more useful behaviour: a filter on such a package selects nothing,
which is a real finding, and reporting it as one beats dying. A broken scan
is still caught twice — a non-zero cargo exit stays fatal, and check() exits
2 if the union across all packages is empty.

21 self-test cases. The two new ones pin both behaviours: a whole-package
step passes with its package absent from the inventory entirely, and a
filter against a package with an empty test set is reported vacuous with the
package named.

Refs #404. REQ-GUARD-GATE-EVIDENCE-002 (c).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@avrabe
avrabe merged commit c28c2cc into main Aug 7, 2026
21 checks passed
@avrabe
avrabe deleted the fix/gate-evidence-batch2 branch August 7, 2026 16:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

check_verification_filters parses the -p package and then ignores it: a cross-package filter reads as non-vacuous (#388 still live)

1 participant